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Background / Context: 

Description of prior research and its intellectual context. 

Due to changing demographics and an increasingly competitive global economy, efforts to 
increase college readiness and matriculation are attracting more attention as a school refonn 
measure. Adolescent students navigate their final years of secondary school making important 
decisions for their continuing education and future careers. The literature on these transitional 
years suggests that adolescents’ pursuit of college and professional careers in Science, 
Technology, Engineering, and Math (STEM) is not deterred by a lack of talent or interest, but 
rather by students’ inability to transfonn their interests into realistic strategies (e.g., course 
selection, targeted extracurricular activities, and college planning) to achieve their career goal. ' 
Findings from these studies have informed the programmatic activities that comprise four of the 
core intervention components of the College Ambition Program (CAP) model: mentoring, course 
counseling and advising, college-related activities and workshops, and teacher professional 
development and instructional support. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

This study is part of a larger project that will test the effectiveness of the CAP intervention 
model beginning with two schools experimental schools and two control schools. The study will 
evaluate the effectiveness of the overall intervention, as well as each of the four specific 
programmatic components. Data will be collected from school records, student surveys, and 
merged with state data. All components of the intervention have specific measures for assessing 
their relative value for increasing college attendance among students who are college ready but 
do not enroll in postsecondary school fall after high school graduation. . 

This smaller study will look at two specific indicants of college readiness in students: their 
ambitions in high school and their test score performance. Ambition is defined as expectations 
of educational attainment and its relationship to occupational aspirations. Performance is 
measured by grades, course taking behaviors, and ACT scores. According to the ACT Condition 
of College and Career Readiness Report for Michigan, students score below the national 
benchmark for college readiness in both math and science. This smaller study will specifically 
focus on the alignment of student’s high school ambitions and performance with their 
postsecondary plans. College plans are derived from an online survey that has been piloted and 
will be administered in the four schools during the month of October. Measures on the survey are 
taken from existing national surveys and some have been modified. A series of new items fielded 
in focus groups for issues of face validity are also included. School transcript information is used 
to obtain course taking behaviors and ACT scores. State information will be used for 
constructing additional matched comparison groups for further analyses. 



1 Schneider & Stevenson, 1999; Czikszentmihalyi & Schneider, 2000; Schneider, 2007, 2008, 2009. 
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Setting: 

Description of the research location. 



CAP is a whole-school intervention that will be implemented in phases. Currently, for the 2010- 
2011 school year, it is being implemented in two public secondary schools — one urban and one 
rural — in Central Michigan. A flagship land-grant university is within 5 miles (10 minutes) of 
the urban high school and 25 miles (30 minutes) of the rural high school. A community college is 
within 1 mile (walking distance) of the urban high school and 20 miles (25 minutes) of the rural 
high school. In addition to the intervention schools, there are two matched — one urban and one 
rural — Central Michigan comparison group schools. For the 2011-2012 year, there will be four 
experimental schools — two urban and two rural with four new controls. All of the schools will be 
followed for three years. All of the schools have lower than average college enrollment rates 
when compared with the state average and with other schools with similar student populations on 
measures of race and ethnicity, socio economic status, and in similar geographic regions. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features or characteristics. 

The urban high school participating in the CAP intervention serves a racially diverse student 
population (39 percent white, 34 percent black, 20 percent Hispanic, 5 percent Asian, and 2 
percent Native American) of approximately 1300 students. The rural school serves slightly less 
than 500 students, nearly all of whom are white. The urban school serves a large percentage of 
economically disadvantaged students, with 60 percent of their students eligible for free and 
reduced lunch. At the rural school, around 30 percent of the students are eligible for free and 
reduced lunch. Both of the schools participating in the CAP model have a substantial number of 
students who would be the first in their family to go to college. The comparison group schools 
have largely similar characteristics as their intervention counterparts. 

Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 

CAP is designed to enhance and supplement existing school resources in order to enhance the 
college-going culture of the school. CAP leverages resources from existing school, community, 
university, and nonprofit partners to create a system of supports for high school students. CAP is 
implemented in phases with the first six-month phase consisting of a needs assessment and the 
introduction of a CAP center near the end of the spring tenn to build interest going into the next 
academic year. The second phase consists of implementing the full intervention based on the 
results of the needs assessment. The CAP chief operating officer works collaboratively with 
current school leadership and teachers to assess the school’s needs in four component areas, 
ensuring that CAP services supplement, not supplant, existing school activities. In this way, CAP 
is more likely to become integrated into the school’s operational structure, helping to ensure 
future sustainability when CAP is run at school level. In addition to the benefits of working 
toward sustainability, being fully integrated into the school increases the chances for positive 
cultural changes. For instance, building strong relationships within the schools allows CAP to be 
integrated as a part of the school-wide plan for student achievement. 
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Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary’ analysis, analytic 
essay, randomized field trial). 

The study is best described as a pretest, posttest, quasi-experimental, comparison-group, 
interrupted time-series design. The study consists of a comparison of students who participate in 
the CAP (intervention) at the two intervention schools versus students in two comparable schools 
who do not participate (comparison). 

This comparison interrupted time-series design is an increasingly common and rigorous 
approach to school reform research when a randomized control design is not feasible (BiFulco 
2002, Bloom 2003). This study incorporates the "short" enhancement to the interrupted time- 
series design, as popularized by MDRC (Bloom 2003). The strategy measures the effect of CAP 
as the resulting difference from the prior pattern of college-going behavior for a specific grade. 
The analysis compares the college-going probabilities of different student cohorts for each high 
school grade using a consistent outcome measure during three baseline years before CAP was 
implemented and at least 3 follow-up years. By comparing follow-up student college-going 
behavior to the baseline years (counterfactual), we obtain valid estimates of the intervention’s 
effects. For this smaller study the focus will specifically be on ambition and college plans with 
achievement as a control and college attendance as the outcome. 

It is assumed that college-going behavior varies randomly across annual student cohorts with no 
clear sign of a systematic increase or decrease over time. This model can be expressed in the 
following equation: Y,- = A + I.D k FYi a + 'LCjXp + v, + e,, (3) where 
Yi = the college-going probability for student i, 

A = a constant estimated from baseline college-going probabilities that is used to project the 
counterfactual for each follow-up year 

Dk = for the student cohort in follow-up year k represents the average deviation of college- 
going probabilities for that cohort from its projected counterfactual, A. Hence, Dk represents 
the refonn’s impact for follow-up year k, 

FYki = equals one if student i was a member of the cohort for follow-up year k and zero 
otherwise 

CjXp = controls explicitly for systematic changes over time in the background characteristics 
(e.g., gender, age for grade, race or ethnicity, special education status, English language 
proficiency, and eligibility for subsidized school meals) of student cohorts 
v, = accounts for cohort effects by adding a random error term (via using random effects 
models as a variance component structure to the impact estimation model), v h for each cohort 
to the random error term, e h for each student, and 
cy = a random error tenn for student i. 

The equation represents the impact of the intervention as its deviation, Dk, from A, where A is the 
same for cohorts in all follow-up years. 

Bloom (2003) described three important advantages to this approach. First, the approach projects 
what student perfonnance for a specific grade in a school would have been without CAP during 
the follow-up years after CAP was implemented. Second, the approach controls for observed 
changes over time in selected student background characteristics by integrating a multiple 
regression model (ZCjXj,) with the baseline projection model (Y = A + YDkFYu + <?,)■ Third, the 
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approach accounts for year-to-year fluctuations in student outcomes due to "cohort effects" by 
integrating a variance component model (v t ) with the regression model (ZCjXjf) and the baseline 
projection model. By including estimates from comparison schools’ deviation from their baseline 
college-going pattern, we protect internal validity by controlling for local history that might have 
affected student college-going behavior while CAP was being implemented at program schools. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

The three primary outcome measures are whether or not students apply to, get accepted to, and 
enroll in college. For students still in high school, the outcome measure of interest will be student 
reported college intentions and predicted probabilities of going to college based on academic 
achievement data (e.g., Michigan Educational Assessment Program, Michigan Merit Exam/ACT 
scores). The study incorporates school administrative data in the fonn of student background 
characteristics and academic achievement for grade 9, grade 10, grade 11, and grade 12 students 
in the intervention and comparison groups schools for each of the three years prior to CAP 
(pretest) and annually during and after CAP implementation (posttest). 

In addition, after CAP is launched in the schools, a student survey is administered annually in the 
early fall to both intervention and comparison group students. In addition to the fall survey, 
seniors are surveyed again in late spring to obtain their post-collegiate plans and summary 
perspectives of their high school and CAP. The student survey is a 40-minute web-based 
questionnaire that collects infonnation from students on their high school experiences as well as 
their postsecondary awareness, aspirations, and planning. Many of the questionnaire items were 
previously used in the Educational Longitudinal Study of 2002 (ELS:2002), which will allow for 
comparison with a nationally representative sample of students. Student identifiers are used with 
the baseline surveys and administrative data to track change over time. 

We measure impacts on three facets of student college-going behavior: (a) average (mean) 
college-going probabilities (or actual rates for students no longer in school), which summarize 
impacts in general — the total improvement per student; (b) the distribution across specific 
ranges, which helps to identify where in the distribution of student college-going probabilities 
impacts were experienced — how different types of students (e.g., likely college-goers versus 
unlikely college-goers) are affected by the intervention; and (c) the variation (standard deviation) 
of proportions, which indicates how the disparity in student college-going probabilities was 
affected — the equity of the intervention. 

We will assess the robustness of our estimates by reviewing alternative statistical models to 
account for: nesting effects at the second (classroom) and at the third (school) level; sample size 
effects (e.g., number of schools, classrooms, and students); and covariate effects (e.g., 
pretreatment measures) (Konstantopoulos 2008). Moreover, we will estimate and contextualize 
the impact of CAP as a national school refonn intervention at scale using a simulation with a 
nationally representative sample of students (Konstantopoulos and Hedges 2006). Here our 
interest is in identifying similar types of public high schools in these national datasets such as the 
Educational Longitudinal Study and the National Household Education Survey. 
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Findings / Results: 

Description of the main findings with specific details. 

At this time, we have preliminary results obtained from a survey given to 12 th graders in the 
spring of 2010 at the two treatment schools. Using these data, we can analyze the relationship 
between a student’s college plans and their level of performance, as measured by the ACT. Prior 
to the CAP intervention, the treatment schools show below average ACT scores when compared 
to Michigan and national averages (see Table 1 in Appendix B). When considering the level of 
college readiness for our schools, we can compare ACT scores with a benchmark score that 
measures the preparation of a student to have at least a 50 percent chance of achieving a grade of 
B or higher in an entry-level credit bearing course (ACT, 2010). Both treatment schools show 
low rates of students who meet these benchmark scores (see Table 2 in Appendix B). At the 
urban school, less than 1 percent of students met all four benchmark scores. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

Initial analyses of the pilot data suggest that the mechanisms in the high school whereby students 
leam information about college are different in rural and urban schools, and are differentially 
associated with student ambitions and plans controlling on ability. We are currently in year one 
of the College Ambition Program, which has the potential to not only aid in better understanding 
the impact of this particular intervention, but also serve as a model for analysis that can be used 
to assess other high school interventions aimed at increasing college-going behaviors. The long- 
term research project aims to show how CAP and similar initiatives can play an important role in 
increasing college- and career-readiness for adolescents by aligning students’ ambitions with a 
realistic strategic plan for obtaining a bachelor’s degree. By the time of the conference, we will 
have a more complete understanding of the student’s plans (including the types of college they 
hope to attend, area of study, and career goals) for the seniors of 201 1 and prior graduation and 
matriculation numbers for the 2009-2010 cohort. This poster will more fully describe these 
differences and what aspects of the intervention appear to be related to student plans and 
ambitions, not only for seniors, but 9 th , 10 th and 1 1 th graders as well. 
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Appendices 

Not included in page count. 
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Appendix B. Tables and Figures 

Not included in page count. 



Table Bl: Average ACT Scores 





Composite 


English 


Mathematics 


Reading 


Science 


National 

Michigan 
Urban School (SE) 
Rural School (SE) 


21 

19.7 
17(2.5) 
19.1 (5.2) 


20.5 

18.9 

15.8(4.7) 
17.9 (4.7) 


21 

19.7 

16.8(5.4) 
19.1 (4.8) 


21.3 

19.7 

17.6 (4.5) 
18.8 (4.4) 


20.9 

19.9 

17.6 (4.3) 
20 (4.2) 



Source: ACT (2010) Sample based on urban school, n=239, and rural school, n= 75. 



Table B2: Percent of Students Taking the ACT who Met College Readiness Benchmark 





ACT Benchmark for College Readiness 


English 

18 


Mathematics 

22 


Reading 

21 


Science 

24 


All Four 
Benchmarks 


National 


66 


52 


42 


29 


24 


Michigan 


56 


33 


40 


25 


19 


Urban School 


36 


9 


22 


7 


0.4 


Rural School 


52 


24 


33 


20 


11 



Source: ACT (2010) Sample based on urban school, n=239, and rural school, n= 75. 
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